Rename @launchstack/search to @launchstack/retrieval and consolidate all retrieval algorithms and tools into it - #371

… documented folders

Every algorithm and tool is now a folder with its own README, index.ts
surface, and colocated tests: algorithms/{bm25,vector,ensemble,rlm,graph,
reranking} and tools/{citation-builder,rag-search-tool}. The RLM, graph,
and ensemble retrievers move in from apps/web/src/lib/tools/rag with two
seams cut on the way: ~/server/db becomes the store client (same engine
connection), and the ensemble's env reads become configureEnsemble(),
called from the app's new composition seam ~/server/rag/ensemble.ts.

Notes retrieval stays app-side (its tables are product schema the engine
store cannot own) and joins the ensemble as an injected leg — resolving
design question Q2 the low-risk way. The Neo4j graph backend keeps using
indexing's graph client; retrieval's lint allowlist gains that one edge
(Q4). The rag-search tool becomes a factory taking an app-supplied
AccessValidator, since document access lives in product schema too.

Both shim layers (lib/tools/rag, server/rag re-exports) are deleted; old
package subpaths (./retrievers, ./reranking, ./citation-builder) survive
one release as aliases. Ensemble searches now log a per-leg breakdown so
a silently dead leg shows in logs, not just a smaller total.

Verified: workspace typecheck, lint at baseline (0 errors / 62 warnings),
retrieval+tools+evidence+conversion+worker+orchestration+indexing suites,
and the full apps/web jest run (2242 passed) including the vector-retriever
and document-creation integration tests against a migrated pgvector 16.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

…o the package

annOptimizer's five ANN variants become named, documented modules under
algorithms/vector/strategies/ (exact scan, ivf, prefiltered, matryoshka,
adaptive hybrid), with ANNOptimizer surviving as the config-driven
dispatcher — documentQ&A's AIQuery/AIChat routes now import it from the
package instead of reaching into another feature's folder. The feature's
private hybridSearch dissolves along the design's lines: the ts_rank
full-text leg into algorithms/bm25/fts.ts, generic Reciprocal Rank Fusion
plus the composed page-level hybrid search into algorithms/fusion/, with
the query embedder injected rather than chosen by the algorithm.

Two of the three independent embedding clients are retired: predictive's
utils/embeddings and all three notes-vertical OpenAIEmbeddings
instantiations now generate through @launchstack/llm's embedding service
(same endpoints, models, and dimensions — the files keep only env-pair
resolution, caching, and error semantics). One embedding transport remains.

Verified: workspace typecheck, lint at baseline, retrieval + tools vitest,
full apps/web jest (2242 passed) against the migrated pgvector database.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

grounded-retrieval relocates from packages/tools — it is a retrieval
policy, not a vertical capability, and holding it in tools forced a
cross-brick import for every consumer. A re-export keeps the old path
alive for brand-voice/company-context/claim-evidence/persona and the
pipelines' architecture rule ("no feature calls the RAG port directly").

The cost-aware RLM search service moves out of documentQ&A wholesale:
its only app dependencies turned out to be compositions of llm-package
primitives (createEmbeddingModel + resolveEmbeddingIndex) and the store
client, so it lands in tools/rlm-search with zero injected seams. The
Q&A services barrel and AIQueryRLM route re-point to the package.

Every tool folder now carries the convention: README, index.ts surface,
implementation beside it.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

…rface

Four golden tests against the migrated pgvector database pin the moved
ensemble's behavior: top-hit ranking on a fixed corpus, the RRF property
that cross-leg agreement beats a single leg's top rank, a liveness check
on the vector leg exactly as the ensemble constructs it, and the
silent-death canary (retrieval on a seeded corpus must never be empty —
the failure mode the design named worst). If a change legitimately
improves ranking, the goldens move in the same commit, with a reason.

Also: the package README now documents the folder-per-algorithm layout
and subpath map, the changeset describes the full consolidation, and the
publish path is verified — all 154 workspace exports load from built
output (check-package-exports).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

…eside the retrieval rename

Conflicts: the CI publish filter (kept retrieval, added
document-conversion-engine), REPOSITORY.md (kept main's aligned table with
the two new bricks, re-applied the retrieval row and the tools-allowlist
rename), and the lockfile (regenerated from the merged manifests).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>